Improving Medical Students’ Awareness of Their Non-Verbal Communication through Automated Non-Verbal Behavior Feedback

The non-verbal communication of clinicians has an impact on patients’ satisfaction and health outcomes. Yet medical students are not receiving enough training on the appropriate non-verbal behaviors in clinical consultations. Computer vision techniques have been used for detecting different kinds of non-verbal behaviors, and they can be incorporated in educational systems that help medical students to develop communication skills. We describe EQClinic, a system that combines a tele-health platform with automated non-verbal behavior recognition. The system aims to help medical students improve their communication skills through a combination of human and automatically generated feedback. EQClinic provides fully automated calendaring and video conferencing features for doctors or medical students to interview patients. We describe a pilot (18 dyadic interactions) in which standardized patients (SPs) (i.e., someone acting as a real patient) were interviewed by medical students and provided assessments and comments about their performance. After the interview, computer vision and audio processing algorithms were used to recognize students’ non-verbal behaviors known to influence the quality of a medical consultation: including turn taking, speaking ratio, sound volume, sound pitch, smiling, frowning, head leaning, head tilting, nodding, shaking, face-touch gestures and overall body movements. The results showed that students’ awareness of non-verbal communication was enhanced by the feedback information, which was both provided by the SPs and generated by the machines.